Failures of GUI Tests on Different Computer Platforms

There has been anecdotal evidence that failures from testing may be dependent upon the software/hardware platform used to test the target software. We have been involved in the research of testing Graphics User Interfaces (GUIs) [1,2,3], and have established that GUI failures do indeed depend upon the computer platform. The objective of this paper is to evaluate the computer platform parameters that are responsible for different failures observed for different computer platforms for the same tests and GUI software. Another factor is that despite thorough testing, failures may not be manifested in any observable way. References [2], [3] showed that without the use of memory tools, 30% and 85% of the failures detected would have been missed, respectively. The CIS approach is used here for GUI testing [1,2,3]; it involves obtaining all responsibilities, i.e., observable effects in the environment of the GUI system produced by use of one or more GUI objects. For each responsibility, the corresponding complete interaction sequences (CISs) are identified; each CIS consists of the sequence and selections of GUI objects that deliver the responsibility. Each CIS is then modeled by a finite-state model (FSM) to generate the tests for the CIS. One approach utilizes the design of the CIS to model the transitions in the FSM; this leads to design tests, which assure that this CIS is implemented as designed. Another utilizes the implementation of the CIS to model the transitions in the FSM; this requires checking every selection in each GUI object in the CIS; the resulting new paths are then added to the design tests, resulting in implementation tests. For brevity, we will only report faults found by implementation tests, as considerably more faults are detected than by design tests. In the CIS testing method, a failure is observed by the tester, and identified as either due to a defect, where the GUI specification is violated, or a surprise, which exhibits an undesirable behavior not addressed in the specification. The set of defects plus the set of surprises will define the set of faults.